VERA 2.0 Architecture by luca-belli · Pull Request #170 · SpringCare/VERA-MH

luca-belli · 2026-07-09T21:56:53Z

Summary

Defines VERA-MH's target architecture — the shape the codebase migrates to as generate.py/judge.py/run_pipeline.py are replaced by a single vera.py CLI — and locks it down in two documents plus a 6-phase (+2 orthogonal) migration plan:

docs/architecture.md — the readable version: system overview, layer model, CLI surface, package responsibilities, data flow, and a phased migration plan with per-phase "Done when" bars.
docs/ARCHITECTURE-SPINE.md — the terse, numbered version: 26 AD-n invariants, each stating what it binds, what it prevents, and the adopted rule. This is what a PR review or an agent cites (AD-12, not a paragraph) when a change touches a rule.
docs/vera-cli-use-cases.md — the CLI/config surface in full: entity vocabulary (u/c/j), naming scheme, config mechanics, six worked use cases.
.github/CODEOWNERS — enforces required review on the handful of files this doc calls "stable interfaces."

No production code changes in this PR — it's the design contract the migration will be judged against.

Why two architecture docs instead of one

architecture.md and ARCHITECTURE-SPINE.md aren't duplicates — they serve different readers and decay at different rates:

	`architecture.md`	`ARCHITECTURE-SPINE.md`
Audience	Humans getting oriented, PR descriptions, onboarding	Agents and reviewers checking one specific rule
Shape	Prose, tables, diagrams, migration narrative	Numbered `AD-n` entries: binds / prevents / rule
Changes when	Anything about the system's story changes	Rarely — spine entries are meant to be cited by ID and stay stable
How you use it	Read top to bottom once	Grep for `AD-10`, read one paragraph

The spine is derived from the prose doc (sources: front-matter points back at it), not written independently — so the two can't silently diverge into two different "sources of truth." Every rule that shows up as prose in architecture.md also exists as a citable AD-n in the spine; the spine is what makes "which invariant does this PR violate" answerable in one line instead of a paragraph search.

Why this justifies its own PR (and its own doc, not just a README section)

This isn't incremental documentation — it's the reason the migration is safe to do at all with heavy agent involvement:

The codebase is explicitly agent-coding-optimized. Most files are fair game for an agent to change freely inside its own package's boundaries. A small, named set of stable interfaces (llm_interface.py, queue.py, role.py, naming.py, config_schema.py, storage_backend.py) require a design doc before modification — called out individually in CODEOWNERS, not just covered by a package-level rule, so their significance is visible at a glance instead of being one file among many in a directory.
Invariants are enforceable, not aspirational. Import boundaries (generate/ ⊥ judge/ ⊥ score/, workers/ never importing domain, utils/ as a true leaf) are meant to be checked by import-linter + grimp in CI, phased in incrementally as each boundary is created — not just prose an agent might rationalize around.
ESCALATE is a hard stop, and documenting a phase doesn't pre-clear it. Changing a stable interface, adding a subcommand, or touching import boundaries requires maintainer sign-off at the moment of the change, even if this doc already described that exact phase. This is the one place a human is guaranteed to be in the loop on the highest-risk edits (e.g. Phase 5's workers/ rewrite).
The migration itself needed a plan a reviewer could hold an agent to. Each of the 6 phases (0, 1–5) plus 2 orthogonal phases (S: storage abstraction, O: OpenSpec adoption) has a concrete "Done when" bar — not "phase looks done," but a specific, checkable condition (e.g. Phase 4: a 2+-rubric config produces separated output and a pre-existing single-rubric config still behaves identically).

Key design decisions worth knowing about

Two independent pipelines, one CLI. Generation (u↔c LLM conversation) and judging (j walks rubric questions) never import each other; vera.py is a thin orchestrator with zero business logic.
Interfaces live with their implementations, not in a shared interfaces/ package. llm_clients/ owns LLMInterface + every provider, workers/ owns QueueProtocol + queue implementations, storage/ owns StorageBackend + backends — grouped by concern (Common Closure Principle), so an LLM-provider change and a storage-backend change never force an unrelated shared package to move.
Rubrics are self-describing bundles, not a bare .tsv path with assumed sibling filenames — --rubric/judging.rubrics[] point at a manifest describing rubric file, prompts, and (informational-only) intended personas.
Traceability is a hash discipline, not a convention. config.json (immutable) + config.json.sha256 (sidecar) + state.json (mutable, resume state) — the sha256 is computed exactly once and reused everywhere it's needed, never independently recomputed.
Rubric navigation lives in code, never in the prompt. QuestionNavigator deterministically walks the rubric's flow data; the judge LLM answers the current question only and never decides what comes next — this keeps evaluation deterministic and testable.
Data stays out of code. Rubric/persona content must live in data/, never as Python constants — a non-developer must be able to add or edit a rubric without touching code.

What's not in this PR

No implementation. generate.py/judge.py/run_pipeline.py are untouched; Phase 0 (rubric bundle manifest helper) and beyond ship as separate, individually-escalated PRs.

emily-vanark · 2026-07-11T00:14:27Z

@@ -0,0 +1,207 @@
+# VERA-MH Architecture
+
+Validation of Ethical and Responsible AI in Mental Health: simulate mental-health conversations, evaluate them against a clinical rubric, and aggregate scores for comparison across provider agents.


maybe "one or more clinical rubrics"?

And "aggregate scores for each rubric for comparioson..."?

emily-vanark · 2026-07-11T01:13:43Z

+
+| Concept | Location | Notes |
+|---------|----------|-------|
+| Persona | `data/personas.tsv` | Simulated patient; drives the user-side LLM |


would probably use "User" as more neutral than "patient".

great point! Luca and I talked about this a while ago and here are our thoughts
#173

emily-vanark · 2026-07-11T01:17:29Z

+| Persona | `data/personas.tsv` | Simulated patient; drives the user-side LLM |
+| Transcript | `p_*/conversations/*.txt` | Turn-by-turn chat log |
+| Rubric | `data/rubric.tsv` | Question flow, dimensions, severity |
+| Evaluation run | `j_*/` | TSV results, logs, metadata |


Are we moving away from this structure? Because... that seemed logical to me.... grouping the conversations together with their evaluations...

Sorry are you referring to this diagram? https://github.com/SpringCare/VERA-MH/pull/170/changes#diff-140eef3ba41bdcf401d507408084181f2c0ac627532b61e0f7906ea7cc926782R19

For now, no major changes in the strture, but we need to reconsider naming, as they are getting a little cumbersome.

emily-vanark · 2026-07-11T01:45:55Z

+
+- Domain packages do not import each other.
+- Infrastructure does not import domain packages.
+- `utils/` is a leaf layer — it does not import domain or infrastructure packages.


What about data files? (I'm thinking about deriving dimension names and rating values from rubrics, which we might need in both judge and score...)

good call out. those are not going to be inside src as I want them to be accessible to everyone. All the files in src can of course reference those

emily-vanark · 2026-07-11T02:05:24Z

+    ├── conversations/
+    │   ├── *.txt
+    │   └── logs/
+    └── evaluations/


Is it possible we want a layer in between evaluations and specific judge runs that groups the judge runs by rubric / evaluator? So evaluations/SI/j_.... vs...

oh good point. Will think this through.
and / or the j_ part contains that kind of info as well

yeah this is interesting!
I wonder if we want an overall parent+timestamp folder
small plug for coolname 😂

something like

output/ └── prophetic-bullfrog-20260713/ | └──── run-config.json │ └──── SI/ │ └────── conversations/ │ │ ├── *.txt │ │ └── logs/ │ └────── evaluations/ │ │ └────── j_.../ │ │ │ ├── *.tsv │ │ │ ├── results.csv │ │ │ ├── scores/ │ │ │ └── logs/ │ │ └────── j_.../ │ │ │ └── ... │ │ └────── j_pooled/ │ │ │ ├── results.csv │ │ │ └── scores/ └──── HFO/ └────── ...

but having some of the config in the name is helpful.. so

output/ └── p_<user>__a_<agent>__t<turns>__r<runs>__<timestamp>/ | └──── run-config.json │ └──── SI/ │ └────── conversations/ │ │ ├── *.txt │ │ └── logs/ │ └────── evaluations/ │ │ └────── j_.../ │ │ │ ├── *.tsv │ │ │ ├── results.csv │ │ │ ├── scores/ │ │ │ └── logs/ │ │ └────── j_.../ │ │ │ └── ... │ │ └────── j_pooled/ │ │ │ ├── results.csv │ │ │ └── scores/ └──── HFO/ └────── ...

tbh i think a lot of that will be driven by testing
and i am trying to get rid of pooled judges but maybe that's impossible

emily-vanark · 2026-07-11T02:06:50Z

+            └── scores/          ← created by vera score
+```
+
+Pooled runs (`j_pooled__.../`) sit alongside `p_*` folders when merging multiple judge outputs. Full naming rules, legacy flat folders, and `output/adhoc` behavior: [README.md](../README.md).


Is this pooling mutliple judges for the same conversations? If so, I think I'd expect it to be at the level of conversations and evaluations within p_*. If it's something else... what is it, please?

I think this is a hack that @jgieringer and I discussed some time ago to make it fit use cases we didnt support. It pools multiple generated convos and uses the same judge

yeah this was something I overlooked.. the pooled judgings by default go to /output instead of the p_... dir 🤦
I would like it to be something like what I put in another comment

output/ └── p_<user>__a_<agent>__t<turns>__r<runs>__<timestamp>/ | └──── run-config.json │ └──── SI/ │ └────── conversations/ │ │ ├── *.txt │ │ └── logs/ │ └────── evaluations/ │ │ └────── j_.../ │ │ │ ├── *.tsv │ │ │ ├── results.csv │ │ │ ├── scores/ │ │ │ └── logs/ │ │ └────── j_.../ │ │ │ └── ... │ │ └────── j_pooled/ │ │ │ ├── results.csv │ │ │ └── scores/ └──── HFO/ └────── ...

emily-vanark · 2026-07-11T02:10:14Z

+
+**Status: draft, open for review.** Please flag anything missing or wrong.
+
+## Use case 1 — End-to-end test of one LLM


... on a single evaluator? On multiple evaluators?

both/either
will specify thanks

reminds me that we want to support multiple user models
... -u gpt-5.4:2 claude-sonnet-5:1 ...

this also reminds me of a bug: with LLMs having different params (reasoning_effort vs thinking_effort), -jep gets applied to all of the -j models
so -j gpt-5.4:2 claude-sonnet-5:3 -jep reasoning_effort=low would try setting reasoning_effort to sonnet and might get an error from claude's api due to unknown param.

Creating an issue to follow-up on this.

thank you, that's very valuable info and will influence the design
i think we should keep the extra parameters as close as possible to their original names to avoid confusion

I'm still not clear, reading this, on the answer to my first question in this thread. I don't see it explicitly saying anywhere, "this will run all the evaluators" or "this will run only one evaluator" or "this will run all the evaluators specified in the run.json config file" or... ? I'm still confused.

emily-vanark · 2026-07-11T02:11:50Z

+vera pipeline --config run.json
+```
+
+**Open question:** should this ever support multiple providers natively (one combined comparison report), or is comparing providers always done by invoking this once per provider (external loop)?


I... think it depends whether we think folks are going to have lots of hyperparamters for the different models, and whether we're sticking with command line arguments instead of a config file. I think it would be nice to be able to easily specify, "run on models X, Y, and Z with these hyperparameters"... but I don't think that's user-friendly with command line args.

As I read more closely... I do see there's a --config run.json here - should we delineate what goes in that at all in the architecture.md?

absolutely. this is the next step after the top level API is finalized. I'll a note to say that the config and the CLI are going to respect the same set of parameters

emily-vanark · 2026-07-11T02:13:37Z

+
+**Open question:** should this ever support multiple providers natively (one combined comparison report), or is comparing providers always done by invoking this once per provider (external loop)?
+
+## Use case 2 — Batch generate across personas


Are these... all personas in a single persona file? Or potentially multiple persona files (and persona prompts) for mutliple evaluators?

Good question, and I will specify. I think multiple files, with the case of single file as well. Each file contains multiple personas. There might be some duplicate personas

This is another one where I don't actually see the answer specified below... I still deeply do not understand what persona file(s) would be run by vera generate gpt:1 sonnet:2...

Or, for the matter, what LLM(s?) would be evaluated by vera generate gpt:1 sonnet:2.... is there a default config that's specifying everything not specified on the command line, even if we don't use --config? Are there a bunch of defaults hidden in the vera.py parser or the files it calls?

Do we need both lines 35 and 36 to make this work? They... don't look like a single command to me, right now. But maybe that's how they're meant?

emily-vanark · 2026-07-11T02:14:38Z

+
+## Use case 3 — Judge existing conversations
+
+Judge one **or more** existing transcript folders against one or more rubrics. Each rubric has a default judge-LLM set, overridable per rubric. Multiple judges per rubric are supported (for judge-agreement analysis).


Do you want to force judges to work only on conversations generated from their intended personas? Or can those be decoupled?

Those can be decoupled. The human user will point to the set of conversations that are required

emily-vanark · 2026-07-11T02:16:37Z

+vera judge --conversations p_run_a/ p_run_b/ --config run.json
+```
+
+**Open question:** when judging 2+ folders together on the same rubrics, is the output one combined comparison result (side-by-side per-folder scores), or independent per-folder `results.csv` (batching as convenience only, no cross-folder aggregation)? This is a separate open question from use case 1's — resolving one does not resolve the other.


I feel like keeping them separate is cleaner and then letting the user concat is kind of trivial. But curious what @jgieringer thinks.

There are also in-between options eg separate, but the scoring layers aggregates them

emily-vanark · 2026-07-11T02:18:08Z

+## Not yet covered — flag if needed
+
+- Dry-run / validate-only (check config and inputs resolve without spending any LLM calls)
+- Resuming a partially-completed run


I think figuring out how to gracefully resume a run is key - @jgieringer spent a lot of time trying to de-pain this process for the VERA v1 codebase. We need it built in here from the start.

I think outputting some sort of run-config.json will be crucial to make this less painless. AND should be a very trivial thing to add.
It could even be something like:
vera resume --config path/to/output/run-config.json

the run-config.json would store the initial run's config plus # of errors, paths things were stored, your api_key (jk), etc.

and given the metadata stored, it "knows" exactly what needs resuming.

emily-vanark · 2026-07-11T02:18:23Z

+
+- Dry-run / validate-only (check config and inputs resolve without spending any LLM calls)
+- Resuming a partially-completed run
+- Listing available personas/rubrics/judges


This seems helpful

Will add thank you

jgieringer

First pass through. This looks great.
I want to gander again tomorrow.
I love having vera this vera that ❤️

This wouldn't be merged now, though, right??

jgieringer · 2026-07-13T22:59:00Z

+## Not yet covered — flag if needed
+
+- Dry-run / validate-only (check config and inputs resolve without spending any LLM calls)
+- Resuming a partially-completed run


I think outputting some sort of run-config.json will be crucial to make this less painless. AND should be a very trivial thing to add.
It could even be something like:
vera resume --config path/to/output/run-config.json

the run-config.json would store the initial run's config plus # of errors, paths things were stored, your api_key (jk), etc.

and given the metadata stored, it "knows" exactly what needs resuming.

jgieringer · 2026-07-13T23:24:34Z

+
+**Status: draft, open for review.** Please flag anything missing or wrong.
+
+## Use case 1 — End-to-end test of one LLM


reminds me that we want to support multiple user models
... -u gpt-5.4:2 claude-sonnet-5:1 ...

this also reminds me of a bug: with LLMs having different params (reasoning_effort vs thinking_effort), -jep gets applied to all of the -j models
so -j gpt-5.4:2 claude-sonnet-5:3 -jep reasoning_effort=low would try setting reasoning_effort to sonnet and might get an error from claude's api due to unknown param.

Creating an issue to follow-up on this.

jgieringer · 2026-07-13T23:28:52Z

+
+**Status: draft, open for review.** Please flag anything missing or wrong.
+
+## Use case 1 — End-to-end test of one LLM


jgieringer · 2026-07-13T23:48:00Z

+
+| Concept | Location | Notes |
+|---------|----------|-------|
+| Persona | `data/personas.tsv` | Simulated patient; drives the user-side LLM |


great point! Luca and I talked about this a while ago and here are our thoughts
#173

jgieringer · 2026-07-13T23:49:25Z

+| Persona | `data/personas.tsv` | Simulated patient; drives the user-side LLM |
+| Transcript | `p_*/conversations/*.txt` | Turn-by-turn chat log |
+| Rubric | `data/rubric.tsv` | Question flow, dimensions, severity |
+| Evaluation run | `j_*/` | TSV results, logs, metadata |


jgieringer · 2026-07-14T00:07:36Z

+    ├── conversations/
+    │   ├── *.txt
+    │   └── logs/
+    └── evaluations/


yeah this is interesting!
I wonder if we want an overall parent+timestamp folder
small plug for coolname 😂

something like

output/ └── prophetic-bullfrog-20260713/ | └──── run-config.json │ └──── SI/ │ └────── conversations/ │ │ ├── *.txt │ │ └── logs/ │ └────── evaluations/ │ │ └────── j_.../ │ │ │ ├── *.tsv │ │ │ ├── results.csv │ │ │ ├── scores/ │ │ │ └── logs/ │ │ └────── j_.../ │ │ │ └── ... │ │ └────── j_pooled/ │ │ │ ├── results.csv │ │ │ └── scores/ └──── HFO/ └────── ...

but having some of the config in the name is helpful.. so

output/ └── p_<user>__a_<agent>__t<turns>__r<runs>__<timestamp>/ | └──── run-config.json │ └──── SI/ │ └────── conversations/ │ │ ├── *.txt │ │ └── logs/ │ └────── evaluations/ │ │ └────── j_.../ │ │ │ ├── *.tsv │ │ │ ├── results.csv │ │ │ ├── scores/ │ │ │ └── logs/ │ │ └────── j_.../ │ │ │ └── ... │ │ └────── j_pooled/ │ │ │ ├── results.csv │ │ │ └── scores/ └──── HFO/ └────── ...

jgieringer · 2026-07-14T00:37:07Z

+            └── scores/          ← created by vera score
+```
+
+Pooled runs (`j_pooled__.../`) sit alongside `p_*` folders when merging multiple judge outputs. Full naming rules, legacy flat folders, and `output/adhoc` behavior: [README.md](../README.md).


yeah this was something I overlooked.. the pooled judgings by default go to /output instead of the p_... dir 🤦
I would like it to be something like what I put in another comment

output/ └── p_<user>__a_<agent>__t<turns>__r<runs>__<timestamp>/ | └──── run-config.json │ └──── SI/ │ └────── conversations/ │ │ ├── *.txt │ │ └── logs/ │ └────── evaluations/ │ │ └────── j_.../ │ │ │ ├── *.tsv │ │ │ ├── results.csv │ │ │ ├── scores/ │ │ │ └── logs/ │ │ └────── j_.../ │ │ │ └── ... │ │ └────── j_pooled/ │ │ │ ├── results.csv │ │ │ └── scores/ └──── HFO/ └────── ...

…gration phasing - Introduce u/c/j entity vocabulary (user/chatbot/judge), replacing the ambiguous p_ prefix and "patient"/"provider" terminology throughout - Formalize config.json/state.json traceability (immutable config + sha256 sidecar, mutable resume state) and the orthogonal generation/judging config split - Add use cases 5 (pool) and 6 (resume), resolve open questions from the PR thread (multi-folder judge output, multi-rubric layout, folder naming, provider-vs-model identifiers) - Split migration into 4 phases: cosmetic wrapper, scoring split, multi-rubric support, substantial refactor - Remove stale "why async judges are not async?" TODO

… policy - Phase 1 is no longer "cosmetic" -- it fully replaces the top-level scripts with the new CLI/config surface (-u/-j/--sample/--config) from day one, using an informal config.json shape until Phase 3 formalizes it as a stable interface - Split scoring-split and multi-rubric-support into separate phases (5 total); multi-rubric no longer depends on artifacts that don't exist until a later phase - Add per-phase "Done when" acceptance bars, incremental import-linter enforcement instead of all-at-once, explicit legacy-output-data and rollback stances - Introduce "stable interfaces" concept: ABCs/Role/naming module/config schema require a design doc to change, enforced via new CODEOWNERS entries -- codebase is optimized for agent coding, so a small set of rarely-changed files get higher scrutiny than the rest

- Phase 0's rubric-loading fix moves to a reusable library helper instead of judge.py's soon-to-be-deleted main(), so Phase 1 reuses it rather than throwing it away - Introduce the rubric bundle manifest format (rubric/prompt files + informational persona links) -- previews the real judging.rubrics[] config shape instead of a throwaway CLI convention - judging.rubrics is a list from day one at every phase (not just Phase 3), closing the same schema-break risk one phase earlier - vera judge explicitly exposes --rubric in Phase 1's CLI surface - Phase 1 requires a parity/regression test suite against the scripts it replaces, and deletes them outright at phase end (no lingering deprecation-stub period, which also removes a MUST-NOT contradiction) - Phase 4's acceptance bar now also verifies single-rubric backward compatibility, not just the new multi-rubric case - Every phase's Done-when now implicitly covers keeping README/AGENTS docs current, stated once instead of per-phase

vera generate drives a multi-turn, non-deterministic LLM conversation -- exact-output parity testing against generate.py isn't meaningful the way it might be for more structural code, and the non-determinism compounds turn over turn. Split Phase 1's testing bar into structural parity (automated, via the existing mock_llm.py harness) and live-LLM manual spot-checks (explicitly not something pytest can certify), so this doesn't get silently treated as fully covered by a single test suite.

…ongs Phase 1 only replaces the CLI front-end -- generate_conversations/judge internals are untouched, so structural parity testing is sufficient and the risk is low. Phase 5's workers/ unification is what actually rewrites the execution engine (both runners move off independent hand-rolled asyncio queues onto the shared workers/ queue/dispatch), which is where concurrency/timing regressions could actually appear. The two-part testing approach (structural parity + manual live-LLM spot-check) now attaches to Phase 5's Done-when instead.

…nfig overlap, provider concurrency, ESCALATE governance - scripts/pool_vera_scores.py's disposition was silently dropped from the original 8-phase plan when it collapsed to 6 phases -- checked again in Phase 5 - Explicitly acknowledge that the Phase 3 naming-scheme swap breaks anything outside vera.py that parses the old p_*/j_* pattern directly - State the separation of concerns between the rubric bundle manifest (what a rubric is) and config.json (how to run it), since they now overlap in subject matter - Flag per-provider concurrency limits as a real Phase 5 requirement -- workers/ enables genuine parallel fan-out with no cap yet - Clarify that documenting a phase here does not pre-clear its ESCALATE requirement; every phase still needs fresh sign-off before it starts

…tries Fixes findings from a code review of PR #170 plus two open Copilot comments: the naming-module contradiction in docs/architecture.md, CODEOWNERS entries pointing at nonexistent files, duplicated/drifted ESCALATE guidance in AGENTS.md, ambiguous "until migration" phrasing, and the generate.py/judge.py vs vera.py framing contradiction in README.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…tion) New storage/ package with a StorageBackend ABC (write/read/exists on raw bytes+keys) plus LocalFilesystemStorage as the default -- same interface-with-implementations idiom already used by llm_clients/ and workers/, so a future non-local backend (S3) is a new implementation, not a rewrite. Explicitly orthogonal to phases 0-5: no dependency on, and nothing depends on it, so it can be pulled earlier or run in parallel if a concrete need shows up. Also fixed a stale generate_conversations/ reference left in CODEOWNERS from the earlier package rename.

rubric-navigation-in-code, non-developer data/ rationale - config.json.sha256 is one canonical computation shared by the run-id folder name and the sidecar -- never two independent values, and never embedded in config.json's own filename - vera pipeline --target <name> is the one deliberate exception where a rubric bundle manifest's personas become authoritative for both generation and judging in one shot; every other invocation keeps generation/judging orthogonal - Rubric navigation (which question comes next) is code-owned (QuestionNavigator), never inferred by the judge LLM from prompt text - Rubric/persona content in data/ is explicitly required to stay outside code, since CLEO must be usable by non-developers

…l phase

emily-vanark · 2026-07-25T00:45:32Z

Remind me why we're merging to main instead of a VERA_2.0 branch?

emily-vanark · 2026-07-25T00:49:25Z

+
+**Resolved:** stays single-chatbot-per-invocation. Comparing chatbots is always an external loop over single-chatbot pipeline runs, consistent with use case 2. Native multi-chatbot support (one combined comparison report) is not built now — flagged as a possible future addition if a real need emerges, not ruled out permanently.
+
+**`--target <name>` shorthand:** `vera pipeline --target SI` resolves `SI` to one rubric bundle manifest (see [Rubric bundle manifest](../architecture.md#rubric-bundle-manifest)) and sets *both* the generation personas and the judging rubric from it in one shot — for the common case of "run the canonical test for X." This is the one deliberate exception to personas/rubrics being chosen independently; every other invocation (`--rubric` plus separately-specified personas, or explicit `--config` blocks) keeps generation and judging fully orthogonal.


So, does this mean that if we don't specifify --target we run... all the evaluators? Just one default evaluator? (which?)

emily-vanark · 2026-07-25T01:02:33Z

+
+```
+vera judge --conversations output/c_sonnet/<run>/conversations/ --config run.json
+vera judge -j claude:1 gpt:2


If line 51 is a standalone call, I don't understand how we know what conversations we're judging, with which rubric.

emily-vanark · 2026-07-25T01:03:51Z

+## Use case 6 — Resume
+
+```
+vera resume --config output/c_sonnet/<run>/config.json


Is this equally valid whether what needs to resume is generate, judge, or score?

emily-vanark · 2026-07-25T01:05:22Z

+
+### `config.json` shape
+
+Top-level `generation` and `judging` blocks are **completely orthogonal** — model selection for one must never influence or be influenced by the other. Each follows the same models-list pattern (a list, not an object keyed by name, so the same model can appear twice with different knobs):


Where is the chatbot to be evaluated specified? In the config.json? If so... where / how? Somewhere else?

emily-vanark · 2026-07-25T01:13:36Z

+
+```bash
+uv run python vera.py pipeline --config run.json
+uv run python vera.py generate -u gpt:1 sonnet:2


what's the -c model here? or does it not need to be specified?

emily-vanark · 2026-07-25T01:16:53Z

+}
+```
+
+Paths are relative to the manifest's own folder. `personas` is **informational only** — it documents which personas this rubric is intended/validated for, for humans and tooling to discover; it does not make generation consume it automatically. Generation still chooses personas independently (the `generation`/`judging` orthogonality invariant holds). This manifest shape is exactly what a `judging.rubrics[]` config entry looks like once Phase 3 formalizes the schema — the format isn't thrown away when the CLI is replaced, it's the design.


So does the personas path have a different "relative to" than the rest of the items in the list? That's... kind of confusing... wonder if there's any way to make it obvious within the manifest? (Does json support comments?)

emily-vanark · 2026-07-25T01:21:36Z

+
+| File | What it stabilizes |
+|------|---------------------|
+| `llm_clients/llm_interface.py` | `LLMInterface` ABC — every provider implements this |


I don't know what ABC means here.

…on, run nicknames, phase testing Addresses Emily's 2026-07-25 re-review round and the earlier RFC comments: add the missing -c/--chatbot selection (CLI, config.json, Domain model), anchor config.json paths to $ROOT instead of CWD (mirroring the manifest's own-folder convention), mirror --target into the input config with its mutual-exclusivity rule, add human-readable run nicknames alongside the sha, split Phase 5's live-LLM testing into an automated tolerance-based smoke tier and a qualitative manual tier, clarify that new rubrics only need Phase 0-4, spell out the design-doc enforcement mechanism (CODEOWNERS + CI gate), and note that judge/score still work against old-layout data via explicit paths even though resume does not.

Fixes two stale/wrong entries (run-id naming lost its nickname segment, AD-23 overclaimed that old-layout data can't be read at all) and adds two new ADs the spine was missing entirely: chatbot selection (AD-27) and config vs. manifest path resolution (AD-28). Extends AD-15 with the concrete design-doc enforcement mechanism and AD-21 with the target field now mirrored into config.json, so every prose rule added to architecture.md and vera-cli-use-cases.md today has a citable AD.

Today's additions (chatbot flag, $ROOT path rule, nicknames, phase testing split) are still design-stage decisions. Flag both docs as subject to revision once implementation surfaces things the design discussion alone couldn't.

Fills the gap between the mechanical live smoke test (did it run, same shape) and the manual spot-check (does it read well): embed old-engine vs new-engine conversations for matching (persona, config) pairs and compare cosine similarity against a calibrated old-vs-old baseline, so content drift is caught statistically without requiring exact wording match or relying on manual review alone.

luca-belli · 2026-07-27T22:58:31Z

Remind me why we're merging to main instead of a VERA_2.0 branch?

This is to show/start enforcing the new rules. Development phases will be added to a new branch

Isolate Human-validated Conversations

…guity generation has two competing LLM roles (chatbot, user); a bare "models" field didn't say which. judging keeps "models" since only one LLM role exists there, so the name is already unambiguous in that block.

Copilot

Pull request overview

Copilot reviewed 49 out of 98 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (7)

docs/vera-cli-use-cases.md:88

This link uses ../architecture.md, but architecture.md is in the same directory (docs/). As written, the link will 404 on GitHub.

`--rubric`/`judging.rubrics[]` entries point at a [rubric bundle manifest](../architecture.md#rubric-bundle-manifest) (canonical definition), not a bare `.tsv` path.

docs/vera-cli-use-cases.md:132

This link target uses ../architecture.md, but the referenced file is docs/architecture.md (same directory). The current relative path will be broken.

- **Path fields inside `config.json` (`generation.personas`, etc.) resolve relative to `$ROOT`** — the directory containing `vera.py` — never relative to the current working directory the CLI was invoked from, and never relative to `config.json`'s own location. This is a single rule regardless of how the config arrives (`--config <file>`, `--config -`, or `VERA_RUN_CONFIG`), so a config's meaning never depends on where your shell happens to be or where you saved the file. This is distinct from the [rubric bundle manifest](../architecture.md#rubric-bundle-manifest), which deliberately resolves relative to *itself* instead, so a manifest folder stays portable across checkouts — `config.json` doesn't need that property, since it's checkout-specific by nature.

AGENTS.md:99

The “Target CLI” examples still use legacy flag names (--user-agent, --provider-agent, -p, -f) that belong to run_pipeline.py/generate.py/judge.py. This conflicts with the target vera.py CLI contract documented in docs/architecture.md and docs/vera-cli-use-cases.md (which uses -c/-u/-j, --conversations, --rubric, or --config). Keeping these examples as-is will confuse contributors during the migration.

# End-to-end pipeline (target)
uv run python vera.py pipeline \
  --user-agent claude-sonnet-4-5-20250929 \
  --provider-agent gpt-4o \
  --runs 1 \
  --turns 10 \
  --judge-model claude-sonnet-4-5-20250929 \
  --max-personas 5

# Generate / judge / score (target)
uv run python vera.py generate -u claude-sonnet-4-5-20250929 -p gpt-4o -t 6 -r 1
uv run python vera.py judge -f output/{YOUR_P_RUN}/ -j claude-sonnet-4-5-20250929
uv run python vera.py score -r output/{YOUR_P_RUN}/evaluations/{YOUR_J_RUN}/results.csv

.github/CODEOWNERS:7

/generate/ is listed as a CODEOWNERS path, but the repository currently has generate_conversations/ (and generate.py) instead. As a result, changes to the current generation implementation won’t be protected by maintainer review as intended.

/llm_clients/ @SpringCare/vera-mh-maintainers
/judge/ @SpringCare/vera-mh-maintainers
/generate/ @SpringCare/vera-mh-maintainers
/run_pipeline.py @SpringCare/vera-mh-maintainers

publication_data/human_validation/README.md:2

This sentence is uncapitalized and uses awkward phrasing (“is comprised of the data utilized”). Consider rewriting for clarity and standard README style.

conversations_analyzed is comprised of the data utilized in the preprint [VERA-MH: Reliability and Validity of an Open-Source AI Safety Evaluation in Mental Health](https://arxiv.org/abs/2602.05088).

docs/vera-cli-use-cases.md:43

The relative link to the rubric bundle manifest points to ../architecture.md, but architecture.md lives in the same docs/ directory as this file. This link will be broken on GitHub.

This issue also appears in the following locations of the same file:

line 88
line 132

**`--target <name>` shorthand:** `vera pipeline --target SI` resolves `SI` to one rubric bundle manifest (see [Rubric bundle manifest](../architecture.md#rubric-bundle-manifest)) and sets *both* the generation personas and the judging rubric from it in one shot — for the common case of "run the canonical test for X." This is the one deliberate exception to personas/rubrics being chosen independently; every other invocation (`--rubric` plus separately-specified personas, or explicit `--config` blocks) keeps generation and judging fully orthogonal. `--target` never selects the chatbot — `-c`/`generation.chatbot` is required regardless of whether `--target` is used.

README.md:432

The PR description says this PR is only an architecture/design contract with no other changes, but the diff also adds a substantial amount of publication_data/human_validation/conversations_analyzed/*.txt data. If that data addition is intentional, the PR description should mention it (or the data should move to a separate PR) to keep review scope clear.

See **[docs/architecture.md](docs/architecture.md)** for the target layer model, invariants, and single CLI orchestrator (`vera.py`). Below is a quick module reference; CLI usage and output layout are in the sections above.

jgieringer and others added 5 commits June 3, 2026 15:35

isolate human-reviewed conversations

7d132e5

elaborate on data used in preprint

9ede7c5

adding new architecture file

85dd6c6

adding new architecture file

937e42c

new architecture for entry-point

29e3c49

luca-belli changed the title ~~adding new architecture file~~ New architecture Jul 10, 2026

emily-vanark reviewed Jul 11, 2026

View reviewed changes

Comment thread docs/architecture.md

emily-vanark reviewed Jul 11, 2026

View reviewed changes

Comment thread docs/architecture.md

emily-vanark reviewed Jul 11, 2026

View reviewed changes

jgieringer reviewed Jul 14, 2026

View reviewed changes

luca-belli added 6 commits July 16, 2026 16:46

luca-belli requested a review from Copilot July 20, 2026 21:59

Copilot started reviewing on behalf of luca-belli July 20, 2026 22:00 View session

luca-belli and others added 6 commits July 20, 2026 17:00

Add ARCHITECTURE-SPINE.md: terse, AD-numbered invariants contract

0235289

Mark ARCHITECTURE-SPINE.md as final

4e03d63

Add Phase O: firm up OpenSpec adoption from a maybe into an orthogona…

80116f1

…l phase

luca-belli marked this pull request as ready for review July 22, 2026 23:56

emily-vanark reviewed Jul 25, 2026

View reviewed changes

Comment thread .github/CODEOWNERS

emily-vanark reviewed Jul 25, 2026

View reviewed changes

Comment thread docs/architecture.md

emily-vanark reviewed Jul 25, 2026

View reviewed changes

luca-belli added 4 commits July 27, 2026 14:30

Add best-guess caveat to architecture.md and the spine

b0e91ea

Today's additions (chatbot flag, $ROOT path rule, nicknames, phase testing split) are still design-stage decisions. Flag both docs as subject to revision once implementation surfaces things the design discussion alone couldn't.

Merge pull request #159 from SpringCare/mv-human-validated-assets

6205236

Isolate Human-validated Conversations

luca-belli changed the base branch from main to feat/VERA_2.0 July 28, 2026 22:25

luca-belli added 2 commits July 28, 2026 15:50

Merge remote-tracking branch 'origin/main' into feat/architure.md

e40e464

luca-belli changed the title ~~New architecture~~ VERA 2.0 Architecture Jul 28, 2026

luca-belli requested a review from Copilot July 28, 2026 22:54

Copilot started reviewing on behalf of luca-belli July 28, 2026 22:54 View session

Copilot AI reviewed Jul 28, 2026

View reviewed changes

		@@ -0,0 +1,207 @@
		# VERA-MH Architecture

		Validation of Ethical and Responsible AI in Mental Health: simulate mental-health conversations, evaluate them against a clinical rubric, and aggregate scores for comparison across provider agents.


		Status: draft, open for review. Please flag anything missing or wrong.

		## Use case 1 — End-to-end test of one LLM


		Open question: should this ever support multiple providers natively (one combined comparison report), or is comparing providers always done by invoking this once per provider (external loop)?

		## Use case 2 — Batch generate across personas


		## Use case 3 — Judge existing conversations

		Judge one or more existing transcript folders against one or more rubrics. Each rubric has a default judge-LLM set, overridable per rubric. Multiple judges per rubric are supported (for judge-agreement analysis).


		Resolved: stays single-chatbot-per-invocation. Comparing chatbots is always an external loop over single-chatbot pipeline runs, consistent with use case 2. Native multi-chatbot support (one combined comparison report) is not built now — flagged as a possible future addition if a real need emerges, not ruled out permanently.

		`--target <name>` shorthand: `vera pipeline --target SI` resolves `SI` to one rubric bundle manifest (see [Rubric bundle manifest](../architecture.md#rubric-bundle-manifest)) and sets both the generation personas and the judging rubric from it in one shot — for the common case of "run the canonical test for X." This is the one deliberate exception to personas/rubrics being chosen independently; every other invocation (`--rubric` plus separately-specified personas, or explicit `--config` blocks) keeps generation and judging fully orthogonal.


		### `config.json` shape

		Top-level `generation` and `judging` blocks are completely orthogonal — model selection for one must never influence or be influenced by the other. Each follows the same models-list pattern (a list, not an object keyed by name, so the same model can appear twice with different knobs):

Uh oh!

Conversation

luca-belli commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why two architecture docs instead of one

Why this justifies its own PR (and its own doc, not just a README section)

Key design decisions worth knowing about

What's not in this PR

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luca-belli commented Jul 9, 2026 •

edited

Loading